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The Problem of Excluded Baseline Students in a Longitudinal Study: 
Correcting National Dropout Estimates and 
Accommodating Eligibility Change Over Time 

Steven J. Ingeis, NORC/University of Chicago 

Paper Presented at the Symposium on NELS:88 of the Longitudinal Research Special 
Interest Group, American Educational Research Association, Annual Meeting, Chicago, 
Illinois, April 6, 1991. 

Abstract 

This paper draws on the experience of NELS:88 in order to examine the implications 
for sample representativeness, national estimation, and for policy studies, of exclusion of 
students who are deemed ineligible by virtue of their inability-stemming from physical, 
mental or linguistic barriers—to participate in studies requiring questionnaire and/or 
cognitive test completion. It also describes a social study undertaken in the NELSs88 First 
Follow-Up to compensate in key respects for the undercoverage bias attendant upon 
exclusion of potential sample members. The special study examines a subsample of the 
ineligible base year students, collects data on their school enrollment status and demographic 
characteristics, and ascertains whether their eligibility status has changed. The special 
foilowback study of Base Year excluded eighth graders will make it possible to calculate a 
more accurate cohort dropout rate for NELS:88 eighth graders between 1988 and 1990, and 
will permit students who have become eligible in the meantime (e.g., through having become 
more proficient in English) to have a chance of re-selection into the study. Change in 
eligibility status must be accommodated by a longitudinal study such as NELSj88, which is 
designed to support nationally representative eighth (1988), tenth (1990), and twelfth grade 
(1992) student samples, as well as maintain the longitudinal eighth grade cohort. 

This discussion is presented in six sections: 

(1) an account of rules and procedures for exclusion of students in the NELS:88 Base 

Year; 

(2) an assessment of how accurately the Base Year exclusion criteria were 

app'ied; 

(3) a discussion of statistical problems associated with excluding students 

from the sample, in particular, both cross-sectional and longitudinal 
undercoverage bias; 

(4) an account of the aims of the NELS;88 First Foliow-Up Foilowback Study 

of Excluded 1988 Eighth Graders; 

(5) an account of the implementation of the Foilowback Study; and 

(6) recommendations concerning ways tc reduce the number of students excluded 

from national studies; and, in terms of correction of estimates and 
accommodation of eligibility status change, the best means of incorporating 
baseline ineligible students in the subsequent waves of longitudinal studies. 
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In section 6, the following specific recommendations are made: 

I. Special means (foreign language translations of questionnaires, one-on-one oral 
administrations, large print versions of survey forms, extended survey administration time) 
should in some cases be adopted, though the cost of special measures— both monetarily and 
in terms of school and individual burden and cooperation— must be carefully weighed against 
the gain in precision of estimates, taking into account the primary research and statistical 
reporting purposes of the study. 

H It is desirable to move toward more common eligibility definitions across major 
nations! studies, to increase the comparability of their findings. Nevertheless, the different 
purposes of these studies must also be taken into account. For example, a study such as 
NAEP will need to exclude certain students who cannot be tested, yet must, by a study like 
NELS:88, be included in terms of their school enrolment status and demographic 
characteristics, in order to generate a nationally precise cohort dropout rate. 

III. The following particular steps should be taken by national education longitudinal 
studies: (a) define eligibility as inclusively as possible; (b) follow ineligible students to 
ascertain change in eligibility status over ti me; (c)gather demographic and school enrollment 
status data on both the excluded portion of the student population, and on all survey 
nonparticipants. 

IV. The strategy of extending population coverage must go hand in hand with efforts 
to reduce unit and item nonresponse and to improve the quality of responses, since the rare 
policy-relevant populations most likely to be excluded are also disproportionately affected 
by measurement error owing to these other factors. 
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Introduction 



in vhe base year of the National Education Longitudinal Study of 1988 (NEL&88), 
students were sampled through a two-stage process. First, stratified random sampling and 
school contacting resulted in the identification of the school sample; second, students were 
randomly selected (with oversampling of Hispanics and Asians) from within cooperating 
schools. 

The target population for the base year comprised all public and private schools 
containing eighth grades in the fifty states and the District of Columbia. Excluded from the 
NELS.-88 school sample are Bureau of Indian Affairs (BIA) schools, special education schools 
for the handicapped, area vocational schools that do not enroll students directly, and schools 
for dependents of U.S. personnel overseas. (For further details of school-level exclusion, see 
Spencer et ai., 1990, p.10). The student population excludes students with severe mental 
handicaps, students whose command of the English language was not sufficient for 
understanding the survey materials (especially the cognitive tests), and students with 
physical or emotional problems that would make it unduly difficult for them to participate 
in the survey. The primary focus of this paper is (1) the consequences of student exclusion 
for the research design and the statistics it will generate, and (2) the special .neasures that 
have been undertaken in NEL&88 to compensate or correct for the effects of exclusion. 
Before either of these two topics is pursued in detail, however, it will be desirable to say 
more about student exclusion in the NEL&88 base year—the 1987-88 school year during 
which the eighth grade cohort was selected and surveyed. 

1. Exclusion of students. To better understand how excluding students with mental 
handicaps, language barriers, and severe physical and e motional problems affects population 
inferences, data were obtained on the numbers of students excluded as a result of these 
restrictions. 

Seven ineligibility codes defining categories of excluded students were employed at 
the time of student sample selections 



A- attended sampled school only on a part-time ^asis, primary 
enrollment at another school. 

B - physical disability precluded student from filling out questionnaires 
and taidng tests. 

C - mental disability precluded student from filling out questionnaires 
and taking tes',3. 

D - dropout: absent or truant for 20 consecutive days, and was not 
expected to return to school. 

E - did not have English as the mother tongue AND had insufficient 
command of English to complete the NELS*88 questionnaires and 
tests. 

? - transferred out of the school since roster was compiled. 
G - was deceased. 



Before sampling, school coordinators— members of the school staff, typically an 
assistant principal or guidance counselor who acted as liaison between the school and the 
study— were asked to examine the school sampling roster and annotate each excluded 
student's entry by assigning one of the exclusion codes. Because eligibility decisions were 
to be made on an individual basis, special education and Limited English Proficiency 
(LEP) students were not to be excluded categorically. Rather, each student's case was 
to be reviewed to determine the extent of limitation in relation to the prospect for 
meaningful survey participation. Each individual student, including LEPs and physically 
or mentally handicapped students, was to be designated eligible for the survey if school 
staff deemed the student capable of completing the NEL&88 instruments, and excluded 
if school staff judged the student to be incapable of doing so. School coordinators were 
told Xhii when there was doubt, they should consider the student capable of participation 
in the survey. Exclusion of students after sampling ("post-roster ineligibles w ) occurred 
either during the sample update just prior to survey day, or on survey day itself. Such 
exclusion after sampling normally occurred because of s change in student status (for 
example, transfer, death). However, in rare instances such exclusions reflected belated 
recognition of a student's pre-existing ineligibility— that is, if an annotation error was 
made and an ineligible student selected for the sample in consequence of such an error, 
ineligibility became apparent later in the survey, whereupon the student was excluded. 



Excluded students were divided into those who were full-time students at the 
school (categories B, C, and E) and those who were not (categories A, u, F, & G). Our 
main concern here is with students who were full-time students at the school but who 
were excluded from the sample. Excluding these students will affect ?stim&tes made 
from the sample. 

Students in categories A (n=329), D (n=733), F (n=3,325), and G (n=6) were either 
not at the school or were present only part time (with primary registration at another 
school, hence a chance of selection into NELS:88 at another school} Thus excluding 
students ir these categories has no implications for making estimates to the population 
of eighth grade students. 

It should be noted that students in category F, those who had transferred out of 
the sampled school, had some chance of being selected into the sample if they transferred 
into another NELS:88 sampled school just as transfers into NELS:88 schools from non- 
NELS:88 schools had a chance of selection at the time of the sample update. The 
sampling of transfer-in students associated with the sample update allowed us to 
represent transfer students in the NELS:88 sample. 

Figure 1 gives the number and percentage of students who fall into each of the 
three exclusion categories (B, C, and E) that may have implications for estimates drawn 
from the base year sample and iubsequent study waves. 

The total eighth grade enrollment for the NELS:88 sample of schools was 202,996. 
Of these students, 10,853 were excluded owing to limitations in their language 
proficiency or to mental or physical disabilities. Thus 5.35 percent of the potential 
student sample (the students enrolled in the eighth grade in the 1,052 NELS:88 schools 
from which usable student data were obtained) were excluded. Less than one half of one 
percent of the potential sample was excluded for reasons of physical or emotional 
disability (.41 percent), but 3.04 percent was excluded for reasons of mental disability, 
and 1.90 percent because of limitations in English proficiency. 



Figure I .--Excluded and non-excluded eighth grade students in NELS:88 base year schools 



Non-cxcliulccI 




N = 202,996 (Total number of eighth grade students enrolled in 1,052 participating schools.) 



Figure 2 

Excluded Students 

NELS:88 Base Year Schools 




N-10,863 
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Put another way (see Figure 2, above), of the 10,853 excluded students, about 57 
percent were excluded for mental disability, about 35 percent owing to language 
problems, and less than 8 percent because of physical or emotional disabilities. Because 
current characteristics and probable future educational outcomes for these groups may 
depart from the national norm, the exclusion factor should be taken into consideration 
in generalizing from the NEL&88 sample to eighth graders in the nation as a whole. This 
implication for estimation carries to future waves. For example, if the overall 
propensity to drop out between the eighth and tenth grades is twice as high for excluded 
students as for non-excluded students, the dropout figures derivable from L.e NEL&88 
First Follow-Up (1990) study would underestimate early dropouts by about ten percent. 

In a school-based longitudinal survey such as NELS:88, excluded students carry a 
second implication for future waves, in addition to their possible impaci on estimation. 
To achieve a thoroughly representative tenth grade (1990) and twelfth grade (1992) 
sample comparable to the High School and Beyond 1980 sophomore cohort (or, for 1992, 
the HS&B 1980 senior cohort and the base year of NLS-72), the follow-up samples must 
approximate those which would have come into being had a new baseline sample 
independently been drawn at either of the later time points, in 1990 (and 1992) one must 
therefore freshen, to give "out of sequence" students (for example, in 1990, those tenth 
graders who were not in eighth grade in the spring of 1988) a chance of selection into the 
study. One must also accommodate excluded students whose eligibility status has 
changed, for they too (with the exception of those who fell out of sequence in the 
progression through grades) would potentially have been selected had a sample been 
independently drawn two years later, and must have a chance of selection if the 
representativeness and cross-cohort comparability of the follow-up sample is to be 
maintained. Thus, for example, if a base year student excluded because of a language 
barrier achieves the level of proficiency in English that is required for completing the 
NELS:88 instruments in 1990 or 1992, that student should have some chance of re- 
entering the sample. A substantial subsample of the base year ineligibles will, 
accordingly, be followed in the NEL&88 first and second follow-ups, to reassess their 
eligibility status and gather information about their demographic characteristics, 
educational paths, and life outcomes. Data on persistence in school to be obtained from 
this subsample will be used to derive an adjustment factor for national estimates of the 
eighth grade cohort's dropout rates between spring of 1988 and spring of 1990, and later, 
between 1990 and 1992. if the excluded student sample is followed after 1992, the 
resulting information will further enhance the generalizability of NEL&88 data in 
determinations of how many dropouts eventually gain high school diplomas or equivalency 
certificates and by what route. 

Of course, this special followbaek study of base year ineligibles does not wholly 
compensate for population undercoverage because students who are educated at home 
or in private tutorial settings, those who are in excluded categories of schools, and those 
who have dropped out of school before reaching the eighth grade, also fall outside the 
NELS:88 base year sample. The size of the pre-Grade 8 dropout population in winter- 
spring 1988 is uncertain. The National Center for Education Statistics has reported that 
12 percent of dropouts ages 16-24 in 1988 had completed six or fewer years of school 
(Frase, 1989). However, over 31 percent of Hispanic dropouts age 16-24 had completed 
only six, or fewer, years of schooling. This finding both confirms thv fact that there is 
a sizable group of students who leave school before entering eighth grade, and suggests 
that the biasing effect of this phenomenon on NELS:88 data may be much more 
pronounced for some subgroups (in particular, Hispanics) than others. 
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2. Statistical Problems Associated with Excluding Students from the Sample. 

Excluded students are a problem for multiple reasons. Excluded students may be 
a source of bias in national estimates both overall and in particular for certain groups. 
This problem appears both cross-sectionally, when one attempts to generalize from 1 988 
NELS;88 base year data; and longitudinally, when one attempts to measure cohort change 
more broadly (for example, to estimate a national dropout rate) or when one tries to 
achieve nationally representative tenth and twelfth grade student samples. 

Cross-Sectional Sample Undercoverage Bias . Estimates for key policy-relevant 
subgroups and sampling strata may be disproportionately affected by ineligibility. 
Exclusion rates vary, geographically and by urbanicity and by school type and by race and 
gender— males tend to be excluded more than f emalesf language minority students (henee 
Asians and Hispanics) and blacks arc excluded at a higher rate than are whites; 
substantially more students are excluded in public schools than in private schools, and 
most particularly in public school systems with large immigrant populations. The impact 
of exclusion on the generalizability of data is well illustrated by the case of students who 
are classified as of limited English language proficiency (LEPs). We estimate that about 
half of the potential LEP students who cculd have fallen into the sample were excluded. 
Indeed, the LEPs who were declared ineligible are precisely those with the more severe 
limitations in English, hence those targeted by intervention programs and of the most 
pressing policy interest. 

Longi tudinal Sample Undercoverage Bias . It should be remembered that NELS:88 
attempts not only to follow a longitudinal sample over time, but to augment the 
longitudinal sample at two time points (1990 and 1992) in order to achieve nationally 
representative tenth and twelfth grade samples. If any of the reasons for exclusion are 
based on individual states that may change over time, the representativeness of the tenth 
and twelfth grade samples will be compromised if 1987-88 eighth graders who have 
overcome their barrier to participation in the meantime are given no chance of 
reselection into the study. 

The potential impact of exclusions on change estimates can be readily illustrated. 
The phenomenon of dropping out prior to school completion offers one of several possible 
examples. Generally, three kinds of dropout statistics are critically important: event 
dropout rates (the proportion of students who leave school prior to completion in a single 
year); status dropout rates (the proportion of the population who have not finished high 
school, viewed at a particular time point); and cohort dropout rates (statistics derived 
from following a cohort over time to determine who drops out, when they dropout, which 
dropouts return to school and when). (For a detailed discussion of types of school 
retention and dropout rates, see Frase 1989). Each of these dropout rate calculations is 
valuable for different purposes; howevt , application of dropout rates for national 
estimation has sometimes been problematic, given the lack of consistency in local and 
state dropout definitions, and the varying quality of the records data available (see, e.g., 
Frase 1989, App.D). The Current Population Survey (a nationally representative cross- 
sectional household study), and longitudinal studies such as High School and Beyond 
(Hj&B), have therefore been of particular importance in obtaining national dropout 
statistics. The modest sample size of the Current Population Survey, however, limits the 
precision of estimates for some of the policy-important minority subpopulations 
(Kaufman and Frase, 1990, p.30). And there are limitations to the HSficB data as well, 
some of which may be overcome by NELS:88. 
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One weakness in the High School and Beyond design was that it began with second 
semester tenth graders; many students drop out before reaching tenth grade. NEL&88 
will correct this design weakness by beginning with eighth graders. HS&B also excluded 
certain categories of students? those who dropped out in the course of tenth grade, those 
with language barriers to participation or with physical or mental barriers to 
participation. These excluded students do not enter into the cohort dropout rate obtained 
from HS&B. NELS:88 will correct this HS&B design weakness too through a followback 
study of excluded NEL&88 base year students. 

NELS:88 employs an event history definition of dropping out (the survey collects 
the time of the event of dropping out) and repeatedly returns to the same individuals to 
update their event histories. It is therefore possible to determine both who drops out and 
who returns to school, and at what stage* and to eventually know what proportion of 
dropouts within the defined cohorts (a representative sample of 1987-88 eighth graders; 
a representative sample of 1989-90 tenth graders) obtain high school diplomas or their 
equivalents. NEL&88 provides both a particularly strong basis for investigating the 
dynamics of school leaving and school completion, as well as a basis for estimating the 
cohort dropout rate for 1987-88 eighth graders. 

However, the probable understatement of dropout rates and loss of 
representativeness to the dropout sample attendant upon the three categories of base 
year student sample exclusions must be taken as a serious consequence of having 
incomplete (95 percent) representation in NELS:88 of the 1987-88 eighth grade 
population. Subgroup estimates may be particularly affected. For example, exclusion 
owing to language barrier may particularly affect groups with high recent immigration 
rates to the United States, including many Hispanic and Asian subgroups. Additionally, 
excluded members of the subgroup are likely to differ in important respects from 
included members. Thus, for example, Bean and Tienda report (1983) that it is precisely 
those Hispanics with lesser English language proficiency who are least likely to complete 
high school. 

The excluded students pose, moreover, a double difficulty for future rounds. First, 
any undercoverage bias introduced in the Base Year will persist in subsequent rounds of 
the study. 

Second, sample freshening 1 alone is not sufficient to ensure a 



In order to compensate for differences between the 1988 eighth grade cohort 
and the tenth and twelfth grade student population frames in 1990 and 1992 —and 
thus ensure full tenth and twelfth grade sample representativeness— two logically 
related subsets of students must be taken account of. First, as noted above, the 
longitudinal cohort of 1988 eighth graders must be augmented by representatives 
of the population of base year ineligible students who change in their 
characteristics such that they uieet NELS:88 eligibility criteria in tenth or 
twelfth grade. Second, one must find the means to sample and collect data from 
the subset of 1990 tenth graders who were not enrolled as eighth graders in 
American schools in 1988. There are, broadly, two types of students who belong 
to this subset, (1) individuals who have not changed grades in "normal 
progression (tho:,e who were held back a grade or skipped a grade) and (2) those 
who were out of the country, such as recent immigrants. The freshening 
population, like the ineligibles population, will prove to have special 



11 



10 



representativetenth grade sample (1990) and twelfth grade sample (1992). If any of the 
bases for exclusion implicate statuses that are potentially transient, then the NEL&88 
First Follow-Up and Second Follow-Up samples will be less than fully representative to 
the extent that excluded categories of eighth graders whose statuses are subject to 
change are given no chance of reselection into the sample. 



3. How Well Were the Base Year Exclusion Criteria Applied? 

It is important to keep in mind that the excluded students were determined by their 
schools to be unable to participate. Criteria for exclusion were provided to the schools, 
but it was up to the school Itself-usueUy the School Coordinator or the principal-to 
interpret and apply the eligibility criteria. Sehools were asked to apply the criteria on 
an individual basis. Thus, LEP students or special education students were not to be 
excluded categorically. Rather, only those particular LEP or special education students 
whose limitations were so severe as to constitute significant barriers to meaningful 
participation were to be excluded. In cases of uncertainty, school personnel were asked 
to include the student. 

A few students were included who manifestly should not have been. Their 
difficulty in completing the questionnaires and tests was noted by survey administrators, 
and Educational Testing Service rejected as unusable a small number (less than one 
percent) of cognitive tests. However, in the main, the extreme cases of physical or 
mental disability, and limitation of English proficiency, were successfully excluded. 
Indeed, one could draw the conclusion that the screening out of students was too 
effective in that one would expect more borderline cases had schools taken with full 
seriousness the injunction "when in doubt, include." 



characteristics and a high density of students from policy-relevant subgroups. 
(For example, not only do Hispanics constitute a significant proportion of 
immigrants, but also, a disproportionate number of Hispanic students fall outside 
the modal grade progression-- De La Rosa and Maw, for example [1990, p. 18] cite 
1988 CPS data that show that "over one third of Hispanic females have fallen one 
or more years behind (34.3Z), as have more than three in seven Hispanic males 
(44. 92)"). The representativeness of the tenth and twelfth grade samples in 
NELS:88 has a double importance. Not only is such representativeness desirable 
to ensure the general izabil ity of cross-sectional findings, but also it wi It 
allow cross-cohort comparisons to be made to the previous NCES studies of high 
school sophomores (HS&B, 1980) and seniors (HS&B, 1980, 1982; NLS-72, 1972). For 
an account of sample freshening in N ELS: 88, see Appendix A. 



Completion rates were in excess of 99 percent for ail tests. Sections were 
not scored if fewer than five items were answered in the section; most students 
in this group answered no items at all. Then a "reasonableness check" was 
performed to identify students with ten or fewer items answered and whose IRT- 
estimated scores were more than three points higher than their raw scores. Most 
deleted cases had zero items answered, and some of these cases could represent 
students who found the tests too difficult to attempt. The percentage of usable 
cases was 99.7 percent in reading and mathematics, 99.5 percent in science, and 
90.2 percent in history/civics/sociai studies. 
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In any case of the application of general criteria, there is bound to be some degree 
of arbitrariness in judgments about borderline eases. This arbitrariness is of course 
compounded when the numbers of people (over a thousand individuals in the NELS:88 base 
year) rendering eligibility judgments is large. Our greatest concern about the 
classification process, however, is that, for reasons of time and burden, some schools 
apparently departed from their instructions an* deluded students on a categorical basis 
in preference to rendering the prescribed case-by-case assessments. (Evidence for this 
phenomenon is seen when sampling rosters are inspected and all students within a pre- 
existing category are excluded.) In c ^sequence of categorical exclusion, one would 
expect that overall, more students may have been excluded than necessary. The 
temptation to exclude categorically— in a school with a large eighth grade, given severe 
time pressures for producing an annotated roster, and with individual-level information 
available to the School Coordinator only through the laborious process of interviewing the 
special education or bilingual education teacher of each student— is large. 

In order to minimize this problem in the future, one solution might be greater 
precision in exclusionary definitions. (This is the direction that NAEP moved in the 1990 
assessment— see Appendix C for the 1990 NAEP criteria-with a modes'*, but important 
impact on the number of ineligibies [the 1990 assessment saw a reduction in t ; .e excluded 
portion of the population of around one half of one percentage point]). Setting out more 
specific conditions for ineligibility would increase school burden and might adver. .4ly 
affect prospects for cooperation in a few cases, but in general would maximize the 
number of participants by minimizing the number of wrongful exclusions. In addition, 
giving schools the option of excluding students from the test while including them for 
purposes of questionnaire administration would further minimize excessive exclusion. 
Both these strategies— further interpreting the broad eligibility criteria, allowing 
questionnaire-only eligibles— nave been pursued in the NELS:88 excluded student 
folio wbaek study that is described below. 



4. The Followback Study of Excluded 1988 Eighth Graders; Aims. 

The longitudinal followback of Base Year excluded students that undertaken in the 
NELSJ88 First Follow-Up and io be repeated in the 1992 Second Follow-Up will help to 
realize several important aims. 

First, it will help to increase the accuracy and generalizability of key population 
inf erences . It will do so uy providing a correction factor for N ELS:88 estimates of school- 
leaving and school completion that encompasses virtually the entire population of 1987- 
88 eighth graders. By checking the school enrollment status of the special pample of 
excluded students, and by gathering additional demographic information, it will be 
possible to generate subgroup-adjusted correction figures for NELS:88 national estimates 
of the rates of remaining in school, dropping out, and dropping out and returning to 
school. 

Second, in cases where an adjustment to estimates cannot be obtained, data from 
the study can serve to qualify estimates , that is, to enter an explicit caveat about their 
limitations. Tnis will be the case, for example, with test results. By gathering basic 
demographic data on who was excluded, it will be known to what extent, both generally 
and for selected subgroups, test results place a probable upward limit on the tested 
achievement of in-school and out-of -school youth in the United States whom the NELS:88 
sample was designed to represent. The biasing effects on test results are likely to be 
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especially severe for selected groups. For example, exclusion of English language non- 
proficient and ; i more severely limited English proficient students presents a biased 
picture of groups with high immigration rates (for example, Koreans), since generally 
recent arrivals—precisely those students nust in need of special assistance and most 
likely to have low scores on a cognitive battery in English—will have been excluded from 
testing. Test results will therefore tend to paint a more optimistic picture of the 
educational progress of certain subgroups than may be justified. Likewise test results 
(and student, parent, school, and teacher data) will be lacking for a portion of the dropout 
population--the portion excluded from the sample. To the extent that excluding students 
lessens tne representativeness of the dropout sample in NELSs88, it is important to know 
as much as possible about the demographic characteristics of excluded 1987-88 students 
who .ave since dropped out of school. 

A third aim that will be served bv the followback study will be to enhance the 
representativeness of the high school-based NELS:88 follt«-up samples . Just as 
freshening is necessary to ensure sample representativeness (freshening gives a chance 
of selection to those 1990 tenth graders and 1992 twelfth graders who are out of 
sequence, that is, were not eighth graders in 1988), so too must one accommodate those 
whose ineligibility status changes over time. (Although technically one should also 
freshen on the excluded students, the monetary cost of doing so would be high and the 
payoff in sampling precision small.) 

A fourth aim that wilt be served by the followback study of excluded eighth 
graders is to correct any errors in the application of past eligibility criteria . We noted 
above that there are probably some erroneous classifications, stemming from categorical 
exclusion of special education or bilingual education students by some schools. Revisiting 
these cases— particularly with a more precisely specifieu interpretation of the original 
eligibility definitions, and access to specialty teachers (bilingual education. v ,lish as 
a second language, special education) who have first-hand knowledge that would 
underwrite accurate individual-level eligibility determinations -would afford an 
opportunity to correct any such misclassificatirns. 

A fi fth and final aim of the study is to enforce consistency in eligibility definitions 
between the base year and follow-ups of NEL&88. bv applying the broadened e Ijgj bili ty 
cri teria of the First Follow-U p (which extended eligibility to students who could 
complete the questionnaire only in Spanish) to excluded 1987-88 eighth graders. 

While demographic and status information should be gathered for all members of 
the excluded student followback sample, there are three situations which justify 
inducting a formerly excluded student into NEL.&88 and administering the student 
questionnaire (and tests, if possible). The three situations are as follows: that person 
{1} has changed, that is, now meets the eligibility criteria; (2) was wrongly classified in 
1988; (3) was rightly classified and the student's limitation is unchanged, but this person 
meets broadened eligibility criteria (that is, can complete the student questionnaire in 
Spanish). 

The ineligibility issue carries with it a special twist to be confronted in the Second 
Follow-' 'p. Some freshened students from the 1990 sophomore sample were deemed 
ineligible, that is, unable to complete the various survey forms and therefore excluded 
from the N ELS.* 88 First Follow-Up. Since the competence of these students may change 
bciv:ccn 1990 and 1992, as may that of Base Year (1988) ineligibles who remained 
ineligible in 1990, the target population for the 1992 follow-back of ineligibles will 
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comprise both the remaining Base Year ineligibles (those not added to the First Follow- 
Up sample in 1990) and the First Follow-Up ineligible* from the freshened samnle. 



5 The Followback Study of Excluded 1988 Eighth Graders: Implementation. 

5.1. Sampling. The sample of ineligible students was drawn from the 1,052 fully 
participating Base Year core schools. Three types of ineligible students were sampled: 
physically handicapped (B), mentally handicapped (C), and language minority (E) students 
with a linguistic barrier to participation. 

Each school folder contained a transmittal which gave the total number of 
ineligibles at each school. The folder also contained a school roster, on which, normally 
the ineligibles were crossed out, with a code entered next to their names to indicate the 
reason for exclusion. After finding the codes for those students, Transfer, Part-Time, 
and Dropout students were eliminated, and the remaining ineligibles sorted by ethnicity. 

All of tne students were numbered consecutively on the roster. If there were 
API (Asian/Pauific Islander) and/or HIS (Hispanic) students, they were numbered 
separately on the roster or on a special form for those groups. Next the ineligibles were 
listed and numbered on a counting form. This procedure was performed for each school. 

The counting form was divided into three sections, NOT (non- API and non-HIS) 
API, and HIS. Students were listed by ethnicity only if it was specified on the school 
transmittal that the school had API or HIS students. Even if the surname was Hispanic 
or AsHn, students were not reclassified if the roster and transmittal did not specify this 
information. (In a few instances ethnicity was not indicated on the roster; these students 
were then listed on the form as NOTs.) The list of students was then numbered, beginning 
with the next consecutive number following the lr-n number used on the roster. For 
instance, if the number of NOTs was 286, the first ineligible NOT would be numbered 287 
and so on. In essence these students were simply added to the bottom of the numbered 
roster. This numbering process was repeated for API and HIS students. 

After the numbering process had been completed for a school, students were 
sampled using the school's original selection table, following the next unused number(s) 
on the table. If there were more students than selection numbers the number(s) were 
imputed for the next selection^). Once the selections were made the names of the 
students were put into a spreadsheet file. There were 10,723 pre-roster ineligibles, of 
whom 1,479 were selected. The next step was to add the post-roster ineligibles to the 
group of pre-roster students? of the 130, 119 were selected. Merging the two files 
produced a sample fra ue of 1,598 students. 

The file was sorted by ?thnicity, eligibility, and pre-roster or post-roster type. 
A serpentine son was then employed. The file was subsampled, using an interval of 
2.37091 and a random start of 1.685831. A total of 674 students were sampled for the- 
NELS:88 Ineligible Study. There are 623 pre-roster Ineligibles and 51 post-roster 
ineligibles. 

These students were included in the pool of ineligibles. Six hundred had originally 
been set a> the target sample size. However, in 172 cases the reason for ineligibility was 
not recorded but the student's name was crossed off the roster. From previous 
information (such as callbacks to schools) we had reason to believe that most of these 
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were transfers, students in a different grade, or students wno were expected but never 
appeared at the school. If so, these students would be eliminated once locating began, but 
it seemed prudent to follow up on these cases to make sure they were transfers or 
students never enrolled (in the school/in eighth grade), and not excluded students. 
Therefore NORC selected 674 = 600/(1- 172/1598) to achieve an ending sample size of 
600. 



5.2 Instrumentation. For all base year ineiigibles in the excluded student followbaek 
sample, the following status information was to be obtained from the student's current 
school (if enrolled) or school last attended (if a dropout) upon screening: 

Sex: male or female; 

Race/ethnicity: white, black, Hispanic, Asian/Pi, American Indian/AN, other 

School enrollment status: dropout - 

more than 20 consecutive unexcused absences between: 

a. March 1, 1989 and March 31, 1990 or 

b. April 1, 1990 and June 30, 1990 

If a student was reported to be a dropout according to the above definition, 
confirmation was then to be obtained from the home. The reason for this is that school 
records sometimes incorrectly describe students who transferred out as dropouts. If the 
home indicates that the student did not drop out but transferred, and in fact is enrolled 
in another school, then further foliow-up would take place with the newly identified 
school. 

Students were next screened for eligibility. (This process is described in 5.3 

below). 

For students classified as still ineligible, no further information was collected, 
beyond locating data to facilitate future follow-up and a detailed description of the 
precise reason for continued (1990) ineligibility. For 3tudents deemed to be eligible, the 
First Follow-Up student questionnaire and new student supplement were administered; 
eligible members of the followbaek sample were asked to answer with the 1989-90 school 
year as their point of reference, to maintain comparability with data collected from the 
main First Follow-Up sample. 



5.3 Data Collection Methodology. Since the need for a followbaek study of base year 
excluded eighth graders was not foreseen at the time that the original design for NELS:88 
was being put into place, the only information collected on ineligible students was their 
name, race/ethnicity (Asian, Hispanic, or Other), and reason for ineligibility. 

Thus a major challenge of the followbaek study— conducted from January to March 
of 1991— was to locate students for whom no locating information was available other 
than the name of the school in which they were enrolled in the autumn of 1987, when 
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NELS:88 sampling rosters were collected. Using this information as a starting point, 
NORC telephone interviewers attempted to trace the excluded student through the 
eighth grade school. When information was not available from this source, the tenth 
grade schools to which the excluded student's peers had dispersed were contacted. Other 
locating resources that would normally be effective in pinpointing the whereabouts of 
adult populations (for exumple, credit bureaux and state Departments of Motor Vehicles) 
were unfortunately not helpful for this population, given that no social security number 
had been collected for student or parent, and given the youth of the excluded sample. 
This group also contained a disproportionate number of mobile students (for example, 
migrants) and students from low-SES families who were less likely to have a telephone 
cr stable address* 

In determining eligibility status in 1990, interviewers were instructed to obtain 
reports from a person with first-hand knowledge of the student. It was not sufficient 
simply to talk to someone in the school office, or the principal. Interviewers were to 
approach the special education teacher, the bilingual education or language arts teacher, 
or other relevant individuals who had first-hand knowledge of the excluded student's 
academic capacities. This process typically entailed talking to multiple staff members 
of the school, until the individual best qualified to assess the student's eligibility status 
was identified. 

Criteria for eligibility and ineligibility were consistent with the broad categories 
used in the base year; however, as in the NAEP 1990 eligibility definitions (see Appendix 
C), these broad categories were given more prec'sion, so that interviewers could offer 
more determinate criteria for interpreting the br :>ad requirements of sample eligibility. 
The revised eligibility criteria employed in the Base Year ineligibles followback appear 
in Appendix D of this paper. 



5.4 Results. The base year ineligibles is just concluding end it would be premature to 
report results at this time. However, when results are reported, there will be several 
points of interest, including: 

the degree of success in finding these base year excluded students, 

the proportion who became eligible for the NELS:88 First Foliow-Up, 

the proportions of those eligible by each ineligibility group (language 
ineligibles are thought to be most likely to change in eligibility status), 

how much reclassification appears to reflect genuine status change; how much 
revision of prior classifications on the basis of more refined eligibility 
criteria; and how much simply the variability attendant upon asking the 
same question twice, for two different groups of evaluators and the 
proportions of the newly eligible, and of those still ineligible, who are 
dropouts or stopouts. 
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6. Implications for Future Studies 

In the concluding section of this paper, we examine the implications of the 
N£LS:88 experience with ineligible students for future education longitudinal studies, and 
make a series of recommendations. First, there are a number of means for further 
reducing—through special measures— the number of students who can be included in such 
studies. However, the costs of special efforts—both monetarily and in terms of school 
and individual burden and cooperation— must be carefully weighed against the added 
precision, taking into account the prime purposes of the study. 

Second , it is desirable to move toward more common eligibility definitions across 
major national studies, to increase the comparability of their findings. Nevertheless, the 
different purposes of these studies must also be taken into account. Taking NAEP and 
NELS:88 as the most obvious example, differences between the two studies argue for 
common definitions of inclusion in the narrow domain of eligibility for NAEP and 
NELS:88 test completion, but for a broader NELS:88 eligibility standard for participation 
at the level of completing the student questionnaire, and for "participation" at the level 
of being included in the expanded sample of individuals for whom school enrollment data 
and demographic data are gathered. 

Third, the following particular steps should be taken by national education 
longitudinal studies: (a) define eligibility as inclusively as possible; (b) follow ineligible 
students to ascertain change in eligibility status over time; (c) gather demographic and 
school enrollment status, through proxy sources, on both the excluded portion of the 
student population, and all survey nonparticipants. 

Fourth, the strategy of extending population coverage must go hand in hand with 
efforts to reduce unit and item nonresponse and to improve the quality of responses, since 
the rare policy-relevant populations most likely to be excluded are also 
disproportionately affected by measurement error owing to these other factors. These 
four topics are discussed at length below. 



6.1 Possible Means for Reducing the Proportion of Excluded Students. Although exclusion 
of students from the NELS:88 sample is a statistical difficulty, it would be an almost 
insuperable operational difficulty to include everyone. The principle of exclusion 
prudently bends a knee to the fact that not everyone is capable of participating in such 
a study and that the costs of including others through extraordinary measures may prove 
prohibitive, when measured against the general purposes and statistical standards of the 
study. Moreover, it would be ethically unconscionable, and futile as an exercise in data 
collection, to have a non-English proficient student, or an educatle mentally retarded 
student, struggle for eighty-five minutes to attempt to complete a cognitive test that the 
student simply could not comprehend. It would be imprudent to ask 8 student who has 
behavior control problems and cannot concentrate for sustained periods to attend a 
three-hour survey session with other students. Nor could schools be expected to 
cooperate with a study that made such demands. Schools are often (and quite rightly) 
unwilling (and sometimes legally constrained) in the matter of allowing students with 
certain handicaps or certified reading levels to participate. 

For these reasons, High School and Beyond, NELS:88, and the National Assessment 
of Educational Progress (NAEP) have all systematically excluded certain categories of 
the school-age population. In general, however, the excluded student problem is 
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somewhat less acute for studies starting in tenth or twelfth grades than for studies such 
as NELS:88 and NAEP, that begin with or include pre-high school populations. NAEP, for 
example, excludes those students who are deemed by tl.e school to be unable to 
participate owing to: no or severely limited English language proficiency, functional 
disability, or mental disability (for example, being classified as educable mentally 
retarded). In its 1988 assessment, NAEP excluded 5.3 percent of eighth graders, 3.7 
percent of twelfth graders, and 6.3 percent of fourth graders. These exclusion rates show 
an increase over 1984 rates (for example, only 3.6 percent of eighth graders were 
excluded in 1984). NAEP's 1988 eighth grade exclusion rate of 5.3 percent, and the 
NELS:88 exclusion rate of just under 5.4 percent, are surprisingly close. Although the 
NELSi88 exclusion rate is not unexpectedly high, one still must ask what further 
measures might be employed to increase the rate of meaningful participation and to thus 
increase the power of survey estimates. 

The 5.37 percent within-school student exclusion rate slightly understates the 
proportion of the eighth grade population excluded from NELS:88 (or NAEP) because 
whole categories of schools are excluded from the sample as well. Specialized schools 
for the handicapped, for example, are excluded from the sample frame. However, these 
schools are not always graded, so that within them it is often difficult to assign 
unambiguous meaning to a concept such as "in the eighth grade." In any event, these 
populations, consisting of the most severely mentally and physically handicapped, would 
be extremely difficult or impossible to survey using the NELS:88 instruments. Reform 
schools and other such institutions also were excluded from the base year school sample, 
as were Bureau of Indian Affairs schools. Thus, even had there been no excluded 
students in the base year, certain students would have been excluded nonetheless because 
their schools would have been deemed ineligible to participate. 

With additional funds, the number of excluded students in NELS:88 or like studies 
could be further reduced by the following means: 

* Translation of questionnaire and tests into Spanish, Chinese, Korean, and other 
languages associated with groups that have a high immigration rate into the United 
States, or otherwise constitute language minority communities (for example, American 
Indians). 

* Extended survey administration time limits for handicapped and language 
minority student* 

* One-on-one oral administration of questionnaires for poor readers and 
handicapped students 

* One-on-one oral administration of questionnaires for NEP and LEP students, 
using bilingual interviewers (Chinese-English, Vietnamese-English, Spanish-English, and 
so on). 

* Large print versions of the instrument for the vis-ially impaired 

Useful as such measures would be, they are costly and would only marginally 
increase the number of students surveyed. Nor are these measures wholly without 
difficulty in concept and application. Some who are non-proficient in English lack 
literacy in their mother-tongue as well; therefore production of multilingual materials 
does not remove the language barrier for all students in these language groups. 
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Production of multilingual written materials must be supplemented by one-on-one oral 
administration to be truly effective. In the case of the student questionnaire, translation 
into (at least some) other languages is feasible, and although there is a question about 
possible mode of administration effects on the comparability of data collected, the 
questionnaire can indeed be readily administered by an interviewer, as is generally the 
case for survey forms designed for self-administration. Production of valid, truly parallel 
multilanguage test forms would be an enormous (and staggeringly expensive) undertaking. 
Moreover, in the case of the cognitive test battery, oral administration of the reading 
test is not possible, and is at least suspect for the other tests, grounded as they are In 
visual formats. 

Despite costs and obstacles to implementing any of the suggested means to 
increasing the proportion of students who can be meaningfully surveyed in NELS:88 and 
like studies, there are reasons for thinking that further investment of resources in this 
area is justified. One step in this direction is already being taken in the NELS:88 First 
Follow-Up. Under the sponsorship of OBEMLA, a Spanish-language questionnaire was 
prepared. Few students in High School and Beyond chose the Spanish-language 
questionnaire option (36/27, 1 1 8 1980 sophomores). Nevertheless, one may speculate that 
the availability of a Spanish translation may influence (in a downward direction) the 
number of students excluded on language grounds by school coordinators, even if many 
of the marginally proficient in English students choose the English version in preference. 
No Spanish language version of the student questionnaire was available for the NELS:88 
Base Year (though a Spanish translation of the parent questionnaire was produced). (Also, 
it is possible that greater numbers of students would utilize a Spanish-language 
instrument in eighth, sixth, or other earlier grades.) 

The fact of a Spanish-language version of the tenth grade student questionnaire 
in 1990 has changed (for freshened students) the eligibility criteria of the study. In the 
Base Year, a student who could not complete the questionnaire in English would have 
been excluded from the sample. In the First Follow-Up, a student who cannot complete 
the questionnaire in English but can do so in Spanish will be included. This Inconsistency 
between the eligibility definitions in the Base Year and First Follow-Up can be rectified 
only in one way. Students who were excluded in the Base Year but who would have been 
eligible under the liberalized ( = can complete a Spanish-language student questionnaire) 
eligibility criteria of the First Follow-Up should be given a new chance of selection into 
the sample. This can be done as part of the followback study of excluded 1988 eighth 
graders. 

6.2. Eligibility in NAEP, versus eligibility in NEL&88. In general, we would urge that 
the major national research studies sponsored by the U.S. Department of Education come 
as close to a common definition of sample eligibility as possible, and consistently take 
measures to assess the biasing effects of sample exclusion. 

However, each study has a somewhat different set of purposes and research 
instruments, and NELSi88 must maintain historical comparability in earlier cohorts 
also. 3 While it is desirable for NAEP and NELS:88 to adopt eligibility definitions that 
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arc highly similar, it nevertheless may be desirable for NAEP to maintain a somewhat 
more exclusive eligibility definition. This is the case because it is commonly felt that 
there are some students who should not be eognitiveiy tested, but are marginally capable 
of completing a survey form such as a student questionnaire (or who can complete a 
questionnaire with special assistance). Also, we argue below (in 6.3) that studies such as 
NELS188 should, for certain purposes (for example, production of national dropout 
statistics) adopt an additional (broader) definition of eligibility and participation, that 
allows for proxy reports about students who cannot complete the survey forms. 
Nonetheless, NAEP and NEL&88 should ideally have the same eligibility criteria for 
completing cognitive tests. 



6. 3 Retaining Excluded Students in a Longitudinal Sample. In general, the best procedure 
for dealing with ineligible students in education longitudinal studies would appear to be 
to gather, from their schools and other source, information about school enrollment 
status, demographic characteristics, and specific handicaps. Such students should 
continue to be followed. Particularly for those students with an initial language barrier 
to participation, it is to be expected that their eligibility status may change over time. 
Eligibility should be reassessed with each wave, and those excluded students who become 
capable of completing the instruments at a later time should be inducted into the sample. 



A direction that we have at least moved toward in NELS188, which represents a 
departure from HS&B, is to acknowledge survey "participation" at two levels. The 
traditional definition of participation is completion of the student questionnaire 
(minimum case), or the student questionnaire and the cognitive test battery (maximum 
case). Nonrespondents—those for whom there is no completed questionnaire— receive no 
final (nonresponse-adjusted) weight and do not appear on the final data file. 

However, an alternative approach is to acknowledge a second level of presence in 
the study, based on whether school enrollment status information can be obtained. 
Particularly for generation of school retention and dropout statistics, and in order to 
statistically accommodate both students who are incapable of participation in the sense 
of test and questionnaire completion and students who are capable but did not 
participate, one should attempt through the school or other proxy sources to obtain basic 
demograpi ic and school persistence data for nonparticipants and ineligibles. A special 
weight can they be created to reflect this expanded definition of the participating survey 
population. 



consistency, so long as those students who enter the sample through changed 
criteria can be removed from cross-cohort (HS&B, NLS-72) comparisons. 

4 

Commonal lty of eligibility rules for cognitive test-taking would facilitate 
cross-walks between NAEP and NELS:88 scales in mathematics and potentially other 
ijbjects. While there are additional issues of comparability of the student 
samples to be resolved if NAEP scales are to be implemented in NELS:88, there is 
no doubt but that a common test eligibility definition would make NAEP-NELS:88 
equating easier to achieve. 
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6.4 Concluding Postscript: Relating Undercoverage to Other Sources of Measurement 
Error. Students who are most likely to be excluded (language minority, especially 
Hispanic; blacks; and particularly black and Hispanic males) are of considerable policy 
interest. Nevertheless, even the members of these subgroups who are included are more 
likely to be survey nonparticipants, are less likely to respond to individual items, and are 
less likely to give valid and reliable reports. In the NELSs88 Base Year, mcle, Asian, and 
Hispanic students tended disproportionately to be nonrespondents, as contrasted to 
female sample members and non- Asian or Hispanic sample members (Ingels, Rizzo, and 
Rasinski, 1989, p.ll). (Whites, blacks, and American Indians were not differentiated on 
the sampling rosters; all three groups fall into the non-Asian non-Hispanic "Other" 
category.) In HS&B, unit (and item) nonresponse has consistently been higher for blacks 
and Hispanics, and for males. 

Among participating NELS:88 eighth graders, males were significantly more likely 
to be item nonrespondents than were females. Also, blacks had the greatest propensity 
toward item nonresponse, followed by Hispanics, Asians, and American Indians (Ingels, 
Rizzo, and Rasinski, 1989, p. 38). In terms of quality of responses, the validity and the 
percentage of cases matched for most NEL&88 base year family background items was 
generally lower for blacks and Hispanics than for whites or Asians (Kaufman, Tuma, 
Rasinski, West, and Lee 1991). In High School and Beyond, (Fetters, Stowe and Owings, 
1984) it was found that data quality (as measured by validity coefficients) was higher for 
females than for males, and that white students provided better quality data than did 
black or Hispanic students. 

These findings suggest a chain of compounding biases. Students who are most 
likely to be excluded typically fall into demographic subclasses in which even the eligible 
students participate at a below-average rate. Those who do participate, however, are 
likely to have a higher rate of item nonresponse. Finally, the quality of the data provided 
is likely to be lower than average. The measurement accuracy that is required to support 
scientific policy analysis and basic educational research is Jeopardized to the extent that 
undercoverage and nonresponse issues are not addressed. While the first step in this 
process is to extend population coverage, efforts must also be made to reduce unit and 
item nonresponse and to improve the quality of responses, with special attention to the 
policy-relevant populations least likely to respond or less likely to respond with maximum 
accuracy. 
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Appendix A: Sample Freshening . 

Sample freshening procedures are employed in order to give some 
chance of selection into the NELS:88 First Fol low-Up (1990) and Second 
Follow-Up (1992) to "out of sequence" students, that is, to 1990 tenth 
graders and 1992 twelfth graders who were not in the eighth grade in the 
1987-88 school year. The specific procedure for sample freshening was 
devised by Martin ft. Frankel as an application of the "half-open 
interval procedure" for minimizing undercoverage. The half-open 
interval procedure has previously been applied primarily in the somewhat 
different context of household surveys to correct for missed dwellings 
in an area frame (see Yates, 1948; and Kish, 1965). However, the 
procedure can be adapted to encompass "missed" (that is, not part of the 
eighth grade cohort because not in eighth grade two years before) tenth 
grade student populations, so that all portions of the population of 
tenth graders in 1990 (and twelfth graders in 1992) can be given a 
chance of selection into the NELS:88 sample. Operational izat ion of the 
freshening procedure for the First Fol low-Up is as followsi 

1. In the fall of 1989, NORC Survey Representatives obtained the 
complete tenth grade student roster from high schools selected 
for inclusion in the sample, in order: 

a) to ascertain that students expected to be enrolled 
are in fact enrolled* and 

b) to perform the student linking ' derations on each 
school's roster that form the basis for 
calculating the selection probabilities for 
"freshened" cases drawn from the tenth grade 
rosters. 

2. Tenth grade 'rosters were examined by the Survey 
Representatives within the school building and each NELS:88 
eighth grade selection present on the roster was linked down 
to the next student (referred to as a "linked" student) on the 
list . 

3. The linked student was "screened" to determine whether he or 
she was enrolled in the eighth grade in the United States two 
years earlier. A review of the names of linked students by a 
school administrator able to consult students' cumulative 
files was sufficient to maae this determination. 

A. Students who were eighth graders in this country two years 
prior are ineligible for the freshened sample. The linking 
process stops with that ineligible student. 

5. Each linked student who was not in the eighth grade in this 
country two years earlier was selected into the "freshened" 
sample component. If a linked student was selected, the 
immediately following student on the list was also considered 
"linked" to the NELS:88 sample member. 
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6. Steps 3, 4, and 5 were repeated for each linked student, with 
the process continuing until a student linked to each NELS:88 
eighth grader was determined to be ineligible. 

A like procedure will be employed for the NELS:88 Second Foilow-Up two 
years later. 
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Appendix B: HS&B BASKL1SE (1980) INELIGIBILITY CRITERIA 

In High School and Beyond, the student sample frame was derived from the 
eligible school sample. (As in NELS:88, certain types of special schools were 
excluded.) Within the eligible school a, a sophomorp was defined as a student who 
expected to complete his/her tenth grade course work between April 1, 1980 and 
August 31, 1980. This was to include those students who might be held back or 
who might repeat tenth grade, but to exclude students dropping out before 
administration of the HS&B questionnaire in the spring of 1980. 

In HS&B, a senior was defined as ; welfth grade student who expected to 
complete his/her high school course work between April 1, 1980 and August 31, 
1980. This group included students who might repeat the grade, as well as "early 
completers" if they were to complete their course work during this time period. 
The twelfth grade cohort, however, was not to include early or late graduates. 

NORC asked e/ich selected school to provide a list of its tenth and 
twelfth grade students, as defined above. All students defined as being 
eligible for sample selection were included in the sampling frame, conversely, 
all students defined as ineligible for sample selection were removed from the 
frame. 

Additionally, however, a student was considered ineligible when that 
student: 

- Was a foreign exchange student 

- Transferred out of the selected school. (A transfer student was 

defined as a student who had left the school and whose 
records were requested for a new school).* 

- Died. 

- Would be unavailable until after Au t 31, 1980* 

- Was listed on the roste* in error. 

~ Had become a drop-out or lost student since he was 

selected. Such a student would have to have been out of 
school fcr 20 or more consecutive days and was not expected 
to return.** 

- Was physically or mentally unable to participate in the survey. *** 



*Transfers~out were not directly replaced. t H*ever, as in NELS:88, all 
transfers-in were given a chance of seK : ion into the sample). All 
other categories above in HS&B base yeat *d to replacement by other 
students from the roster. (HS&B substituted students for the following 
casesr dropout, listed in error, language barrier, too tU [mentally, 
physically], in jail, unavailable entire field period, expelled, find 
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deceased.) No substitution of students was done in the NELS:88 base 
year. 



^Dropouts meet the 20 consecutive days criterion, are at least 16 years of age, 
and are not expected to return to school. Lost students are dropouts in 
all respects except that they are not 16 years of age. 

**While this category was used to cover linguistic exclusion also, a Spanish 
language version of the questionnaire was provided so that students whose 
primary language competence was in Spanish would not be excluded. 
However, only 36 sophomores and 8 1980 HS&B seniors elected to complete 
the instrument in Spanish. 



Appendix C: NAEP 1990 exclusion criteria. 



NATIONAL ASSESSMENT OF EDU CA TIONAL PROGRESS 
CRITERIA FOR EXCLUDING STUDENT FROM THE ASSESSMENT 

THE INTENT IS TO ASSESS ALL SELECTED STUDENTS. THEREFORE, ALL 
SAMPLED STUDENTS WHO ARE CAPABLE OF PARTICIPATING IN THE 
ASSESSMENT SHOULD BE ASSESSED 

Some students may be incapable of participating meaningfully in the ar^essment 
btcause of limited English proficiency or a physical or mental handicap. The 
Local Administrator, with the advice of other staff members, may exclude from the 
assessment only those students who are incapable of taking the assessment 
because; 

The student is a native speaker of a language other than English 
and has been enrolled in an English-speaking public school (not 
including a bilingual education program) for less than two 
consecutive years; 

OR 

The student is a special education student with a Individualized 
Education Plan (IEP) who is mainstreamed less than 50 percent of 
the time in acudemic subjects and the IEP team has determined that 
the student is unable to be assessed. 

Ltudents with limited Eng'ish proficiency and students with IEP's should be 
assessed, if in the judgment of school rtaff, they are capable of taking the 
assessment. When there is doubt, include the student. 
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Appendix D; N ELS: 88 First Follow-Up. Eligibility Criteria for t he Base 
Year IneHgibles Study . 



The following instructions for the interpretation of eligibility were f ' ven to 
data collectors for the followback study of excluded 19P8 eighth graders: 

ELICIBIUTY CRITERIA: 

It is the intention of NELS:88 to include all sample members who are 
capable of meaningful participation in the regular survey under normal survey 
conditions. Therefore, UNLESS there are severe mental or physical handicaps or 
language barriers and the sample members are not capable of completing the survey 
instruments under normal circumstances of survey administration, the student 
should be considered ELIGIBLE for NELS:88; in cases where there is doubt of 
eligibility status, the sample member should be considered eligible. 

It is extremely important that an individual assessment be made of each 
r.ospective sample member's ability to participate. Sample members are NOT to 
be considered ineligible categorically (for example, by virtue of being a special 
education student, or LEP student), since some special education students and 
many limited English-proficient students will be capable of completing the 
NELS:88 questionnaire or questionnaire and tests. In fact, as long as the sample 
member is capable of completing the NELS:88 Questionnaire, even if s/he is 
incapable of completing the N£LS:88 Cognitive Test Battery , the sample member 
should be considered eligible for NELS:88. 

Eligibility status will be determined by the school that the sample 
member attended during the 1989-90 school year (or last attended, in the case of 
dropouts). A knowledgeable official from this school must determine if a sample 
member is capable of participating in the regular NELS:88 survey (i.e., a 
bilingual education/ESL/ianguage arts teacher, or member of an Individualized 
Education Plan team, or o'her knowledgeable school staff familiar with the sample 
member's performance. In order to assist schools in making eligibility 
determinations, we urge that the fcHowing guidelines be followed: 

i. Limited English Proficiency (LEP) and Mo English Proficiency (HEP) students. 

A non-native speaker of English who has been in an English-language 
course of study less than two years may be considered ineligible if the school 
determines that the sample member would not be capable of completing a NELS:88 
questionnaire; this study assumes that, normally, sample members who have been 
enrolled in an English-language course of study for at least two years should be 
considertri eligible for NELS:88. 

However, a sample member who is a native speaker of Spanish who is 
literate in Spanish should b included, even if that student is an NEP. Any 
native speaker of Spanish should be considered eligible for NELS:88 if s/he is 
capable of completing either the English language OR the Spanish- language version 
of the NELS:88 Questionnaire. 
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2. Special Education students. 

To be considered a Special Education student, the student should normally 
have an Individualized Education Plan (IEP). A student with an IEP (or dropout 
with an IEP before dropping out)~-who is not mains treamed in English/ language 
arts; AND who is judged by the school to NOT be capable of completing the NELS:88 
Questionnaire should be considered ineligible for NELS. 

To guide schools in making this determination , we recommend that a sample 
member be judged incapable of completing the NELS:88 Questionnaire if that sample 
member READ AT A GRADE LEVEL FOUR OR MORE LEVELS BELOW THE TENTH GRADE NORM 
during the 1989-90 school year in English (or Spanish)* This may be determined 
by looking at school records such as standardized test scores , or, particularly 
when these are not available, by the judgment of the language arts teacher, IEP 
team members, or other school staff familiar with the sample member f s 
performance* 

3. Students with behavioral disorders or severe physical impairments. 

A sample member who is not normally subject to testing or surveying by 
the school, owing to behavior disorders, severe cognitive deficits, or severe 
ph y s i ca 1 impa i rment s which do not permit the testing/surveying of the sample 
member under any but extraordinary conditions, should be considered ineligible 
tor NELS.-88. 

"Extraordinary conditions" should be defined in relation to departure from normal 
testing and survey protocols. That is, if the sample member would require oral 
rather than self-administration of the survey instruments, large print or braille 
versions of the survey forms, translations other than in Spanish, or other 
extraordinary special assistance or aiis, that sample member should be deemed 
ineligible for participation in NELS;88. 
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